Skip to content

Conversation

@stephen-shelby
Copy link
Contributor

@stephen-shelby stephen-shelby commented Sep 29, 2025

Why I'm doing:

Currently, when optimizer rules encounter bugs, users often see cryptic error messages like Error 1064 (HY000): only found column statistics: {353: sum, 3} without knowing which specific rule caused the problem. This makes troubleshooting extremely difficult in production environments, as users cannot identify the problematic rule to apply workarounds.

When optimizer rules fail, especially during complex query optimization phases, the error messages lack context about which transformation or validation rule triggered the failure. This creates significant challenges for both users and developers in diagnosing and resolving optimizer-related issues.

What I'm doing:

This PR introduces a new session variable enable_optimizer_rule_debug to provide detailed debugging information when optimizer rules fail. The enhancement includes:

New Session Variable: enable_optimizer_rule_debug

  • When enabled, provides detailed error context including the specific rule that caused the failure
  • Shows before/after query plans when rule validation fails
  • Helps identify problematic optimizer rules for troubleshooting
  • Works in conjunction with the cbo_disabled_rules feature from PR #64269 to provide users with immediate workarounds

Usage Example:

-- Enable optimizer rule debugging
SET enable_optimizer_rule_debug = true;

--Rerun the failed query. When a rule fails, you'll see detailed output like:

Optimizer rule debug: Plan validation failed after applying rule [TF_PUSHDOWN_AGG_GROUPING_SET].
Validation error: Invalid plan:
LOGICAL_REPEAT
->  LogicalCTEConsumeOperator{cteId='1', limit=-1, predicate=null}
Input dependency cols check failed. 
The required cols {4} cannot obtain from input cols {13,14,15}.
Hint: This error was caught by enable_optimizer_rule_debug=true
Before:xxx
After:xxx

-- Combined with cbo_disabled_rules for workarounds:

SET cbo_disabled_rules = 'TF_PUSHDOWN_AGG_GROUPING_SET';  -- Disable the problematic rule

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 4.0
    • 3.5
    • 3.4
    • 3.3

@stephen-shelby stephen-shelby requested a review from a team as a code owner September 29, 2025 08:23
@stephen-shelby stephen-shelby changed the title [WIP][Enhancement] support plan validate debug [Enhancement] support plan validate debug Sep 29, 2025
@stephen-shelby stephen-shelby marked this pull request as draft October 20, 2025 14:12
@stephen-shelby stephen-shelby marked this pull request as ready for review October 20, 2025 18:24
@stephen-shelby stephen-shelby force-pushed the test_optimizer branch 4 times, most recently from 26fc032 to 5b4e270 Compare October 21, 2025 04:11
@stephen-shelby stephen-shelby changed the title [Enhancement] support plan validate debug [UT] support plan validate debug Oct 21, 2025
@stephen-shelby stephen-shelby force-pushed the test_optimizer branch 3 times, most recently from 9a28a08 to 50c87ab Compare October 21, 2025 06:06
@stephen-shelby stephen-shelby changed the title [UT] support plan validate debug [Enhancement] Add enable_optimizer_rule_debug session variable for rule failure diagnosis Oct 21, 2025
@sonarqubecloud
Copy link

@github-actions
Copy link

[Java-Extensions Incremental Coverage Report]

pass : 0 / 0 (0%)

@github-actions
Copy link

[FE Incremental Coverage Report]

pass : 45 / 48 (93.75%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/sql/optimizer/validate/InputDependenciesChecker.java 1 3 33.33% [213, 222]
🔵 com/starrocks/sql/optimizer/validate/PlanValidator.java 26 27 96.30% [107]
🔵 com/starrocks/sql/optimizer/base/ColumnRefSet.java 1 1 100.00% []
🔵 com/starrocks/qe/SessionVariable.java 4 4 100.00% []
🔵 com/starrocks/sql/optimizer/task/RewriteTreeTask.java 6 6 100.00% []
🔵 com/starrocks/sql/optimizer/QueryOptimizer.java 3 3 100.00% []
🔵 com/starrocks/sql/optimizer/task/TaskContext.java 3 3 100.00% []
🔵 com/starrocks/sql/optimizer/RowOutputInfo.java 1 1 100.00% []

@github-actions
Copy link

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

@stephen-shelby stephen-shelby enabled auto-merge (squash) October 23, 2025 07:30
@stephen-shelby stephen-shelby merged commit 6a318c2 into StarRocks:main Oct 23, 2025
75 of 76 checks passed
@github-actions
Copy link

@Mergifyio backport branch-3.3

@github-actions
Copy link

@Mergifyio backport branch-3.5

@github-actions
Copy link

@Mergifyio backport branch-3.4

@github-actions
Copy link

@Mergifyio backport branch-4.0

@mergify
Copy link
Contributor

mergify bot commented Oct 23, 2025

backport branch-3.3

✅ Backports have been created

@mergify
Copy link
Contributor

mergify bot commented Oct 23, 2025

backport branch-3.5

✅ Backports have been created

@mergify
Copy link
Contributor

mergify bot commented Oct 23, 2025

backport branch-3.4

✅ Backports have been created

@mergify
Copy link
Contributor

mergify bot commented Oct 23, 2025

backport branch-4.0

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Oct 23, 2025
…le failure diagnosis (#63693)

Signed-off-by: stephen <[email protected]>
(cherry picked from commit 6a318c2)
mergify bot pushed a commit that referenced this pull request Oct 23, 2025
…le failure diagnosis (#63693)

Signed-off-by: stephen <[email protected]>
(cherry picked from commit 6a318c2)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/QueryOptimizer.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/task/RewriteTreeTask.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/task/TaskContext.java
#	fe/fe-core/src/test/java/com/starrocks/sql/plan/GroupingSetsTest.java
mergify bot pushed a commit that referenced this pull request Oct 23, 2025
…le failure diagnosis (#63693)

Signed-off-by: stephen <[email protected]>
(cherry picked from commit 6a318c2)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/QueryOptimizer.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/task/RewriteTreeTask.java
#	fe/fe-core/src/main/java/com/starrocks/sql/optimizer/task/TaskContext.java
#	fe/fe-core/src/test/java/com/starrocks/sql/plan/GroupingSetsTest.java
mergify bot pushed a commit that referenced this pull request Oct 23, 2025
…le failure diagnosis (#63693)

Signed-off-by: stephen <[email protected]>
(cherry picked from commit 6a318c2)
wanpengfei-git pushed a commit that referenced this pull request Oct 25, 2025
…le failure diagnosis (backport #63693) (#64488)

Signed-off-by: stephen <[email protected]>
Co-authored-by: stephen <[email protected]>
Co-authored-by: stephen <[email protected]>
wanpengfei-git pushed a commit that referenced this pull request Oct 25, 2025
…le failure diagnosis (backport #63693) (#64485)

Signed-off-by: stephen <[email protected]>
Co-authored-by: stephen <[email protected]>
Co-authored-by: stephen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants